Stop sync on a null value in a cursor column #19889

rodireich · 2022-11-29T19:26:11Z

What

This change will cause a sync to stop if one of the streams contains a column designated as a cursor that contains a null value.

How

This is adding another step to the verification at the beginning of a read in source postgres.
We look for each stream's cursor column and if that column is NULLABLE, we look for a null value .

Because the lookup can be expensive on a large table, we are only going to do that if the column allows null values.
Otherwise it will immediately pass this test.

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

⚠ Sources (25)

Connector	Version	Changelog	Publish
`source-alloydb`	`1.0.17`	✅	✅
`source-alloydb-strict-encrypt`	`1.0.17`	✅	🔵 (ignored)
`source-bigquery`	`0.2.3`	✅	✅
`source-clickhouse`	`0.1.14`	✅	✅
`source-clickhouse-strict-encrypt`	`0.1.14`	✅	🔵 (ignored)
`source-cockroachdb`	`0.1.18`	✅	✅
`source-cockroachdb-strict-encrypt`	`0.1.18`	✅	🔵 (ignored)
`source-db2`	`0.1.16`	✅	✅
`source-db2-strict-encrypt`	`0.1.16`	✅	🔵 (ignored)
`source-dynamodb`	`0.1.0`	✅	✅
`source-jdbc`	`0.3.5`	⚠ (doc not found)	⚠ (not in seed)
`source-mongodb-strict-encrypt`	`0.1.19`	⚠ (doc not found)	🔵 (ignored)
`source-mongodb-v2`	`0.1.19`	✅	✅
`source-mssql`	`0.4.25`	✅	✅
`source-mssql-strict-encrypt`	`0.4.25`	✅	🔵 (ignored)
`source-mysql`	`1.0.15`	✅	✅
`source-mysql-strict-encrypt`	`1.0.15`	✅	🔵 (ignored)
`source-oracle`	`0.3.21`	✅	✅
`source-oracle-strict-encrypt`	`0.3.21`	✅	🔵 (ignored)
`source-postgres`	`1.0.31`	✅	✅
`source-postgres-strict-encrypt`	`1.0.31`	✅	🔵 (ignored)
`source-redshift`	`0.3.15`	✅	✅
`source-scaffold-java-jdbc`	`0.1.0`	⚠ (doc not found)	⚠ (not in seed)
`source-snowflake`	`0.1.26`	✅	✅
`source-tidb`	`0.2.1`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

rodireich · 2022-11-29T22:26:15Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3578522892
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3578522892
🐛 https://gradle.com/s/suewiodej4r4k

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-11-29T23:27:17Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3578872975
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3578872975
🐛 https://gradle.com/s/3oobqcvwvhy44

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-11-30T03:00:09Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3579927590
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3579927590
🐛 https://gradle.com/s/unczakb73gppk

Build Failed

Test summary info:

Could not find result summary

rodireich · 2022-11-30T05:17:44Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3580553589

rodireich · 2022-11-30T06:29:53Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3580900217
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3580900217
🐛

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 26.05s ==================

rodireich · 2022-12-06T17:06:14Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3631734278

…r-value-is-null

rodireich · 2022-12-06T17:19:11Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3631840630
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3631840630
🐛 https://gradle.com/s/jkabuxotu5lx2

Build Failed

Test summary info:

Could not find result summary

…r-value-is-null

rodireich · 2022-12-06T18:53:13Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3632535673

rodireich · 2022-12-06T19:14:12Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3632695040
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3632695040
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 37.53s ==================

…r-value-is-null

rodireich · 2022-12-06T21:12:55Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3633515740
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/3633515740
Python tests coverage:

	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     208     92    56%   36, 42-44, 49, 54, 77, 83, 89-91, 110, 115-117, 123-125, 131-132, 137-138, 143, 149, 158-167, 173-178, 193, 217, 248, 254, 262-267, 275-280, 288-301, 306-312, 319-330, 337-353
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       133     33    75%   24-27, 46-47, 50-54, 57-58, 73-75, 78-80, 83-85, 88-90, 93-95, 124-125, 159-161, 208
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1599    332    79%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/source_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
================= 14 passed, 5 skipped, 21 warnings in 26.07s ==================

rodireich · 2022-12-06T21:50:12Z

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/3633761606

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

rodireich · 2022-12-06T22:41:34Z

/publish connector=connectors/source-postgres-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3634096892

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Stop sync on a null value in a cursor column

43e6bf5

rodireich linked an issue Nov 29, 2022 that may be closed by this pull request

Database Sources: Handle when a cursor value is NULL #17150

Closed

octavia-squidington-iv added the area/connectors Connector related issues label Nov 29, 2022

rodireich temporarily deployed to more-secrets November 29, 2022 19:26 Inactive

octavia-squidington-iv added connectors/source/postgres connectors/source/jdbc connectors/source/relational-db labels Nov 29, 2022

rodireich temporarily deployed to more-secrets November 29, 2022 19:27 Inactive

Fix quoting in query to solve a failing test

3d85e67

rodireich temporarily deployed to more-secrets November 29, 2022 22:28 Inactive

Fix another escaping issue in query

e6150e3

rodireich temporarily deployed to more-secrets November 29, 2022 23:28 Inactive

Fix failing test

d567f6b

rodireich temporarily deployed to more-secrets November 30, 2022 03:02 Inactive

Fix failing tests

f47a713

rodireich temporarily deployed to more-secrets November 30, 2022 05:19 Inactive

rodireich marked this pull request as ready for review November 30, 2022 07:17

rodireich requested a review from a team as a code owner November 30, 2022 07:17

rodireich requested a review from VitaliiMaltsev November 30, 2022 07:18

rodireich temporarily deployed to more-secrets December 6, 2022 05:52 Inactive

Merge branch 'master' into 17150-database-sources-handle-when-a-curso…

09b3d2e

…r-value-is-null

rodireich temporarily deployed to more-secrets December 6, 2022 17:20 Inactive

rodireich temporarily deployed to more-secrets December 6, 2022 17:21 Inactive

Merge branch 'master' into 17150-database-sources-handle-when-a-curso…

7dd7b78

…r-value-is-null

rodireich temporarily deployed to more-secrets December 6, 2022 18:54 Inactive

rodireich temporarily deployed to more-secrets December 6, 2022 18:55 Inactive

Fix failing test

0641b56

rodireich temporarily deployed to more-secrets December 6, 2022 19:15 Inactive

rodireich temporarily deployed to more-secrets December 6, 2022 19:16 Inactive

Merge branch 'master' into 17150-database-sources-handle-when-a-curso…

00bbe1c

…r-value-is-null

rodireich temporarily deployed to more-secrets December 6, 2022 21:14 Inactive

rodireich temporarily deployed to more-secrets December 6, 2022 21:51 Inactive

auto-bump connector version

46c341e

octavia-squidington-iii temporarily deployed to more-secrets December 6, 2022 22:41 Inactive

rodireich merged commit b5c08ce into master Dec 6, 2022

rodireich deleted the 17150-database-sources-handle-when-a-cursor-value-is-null branch December 6, 2022 23:05

octavia-squidington-iii mentioned this pull request Dec 7, 2022

Bump Airbyte version from 0.40.23 to 0.40.24 #20206

Merged

subodh1810 mentioned this pull request Dec 8, 2022

Database Sources: Handle when a cursor value is NULL #17150

Closed

prateekmukhedkar mentioned this pull request Mar 1, 2023

DB Sources MySQL: Handle when a cursor value is NULL #23632

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stop sync on a null value in a cursor column #19889

Stop sync on a null value in a cursor column #19889

rodireich commented Nov 29, 2022 •

edited

Loading

rodireich commented Nov 29, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Nov 29, 2022 •

edited

Loading

⚠ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

rodireich commented Nov 29, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 29, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 30, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 30, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 30, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

Stop sync on a null value in a cursor column #19889

Stop sync on a null value in a cursor column #19889

Conversation

rodireich commented Nov 29, 2022 • edited Loading

What

How

Recommended reading order

rodireich commented Nov 29, 2022 • edited by github-actions bot Loading

Build Failed

github-actions bot commented Nov 29, 2022 • edited Loading

Affected Connector Report

⚠ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

rodireich commented Nov 29, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Nov 29, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Nov 30, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Nov 30, 2022 • edited by github-actions bot Loading

rodireich commented Nov 30, 2022 • edited by github-actions bot Loading

Build Passed

rodireich commented Dec 6, 2022 • edited by github-actions bot Loading

rodireich commented Dec 6, 2022 • edited by github-actions bot Loading

Build Failed

rodireich commented Dec 6, 2022 • edited by github-actions bot Loading

rodireich commented Dec 6, 2022 • edited by github-actions bot Loading

Build Passed

rodireich commented Dec 6, 2022 • edited by github-actions bot Loading

Build Passed

rodireich commented Dec 6, 2022 • edited by github-actions bot Loading

rodireich commented Dec 6, 2022 • edited by github-actions bot Loading

rodireich commented Nov 29, 2022 •

edited

Loading

rodireich commented Nov 29, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Nov 29, 2022 •

edited

Loading

rodireich commented Nov 29, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 29, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 30, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 30, 2022 •

edited by github-actions bot

Loading

rodireich commented Nov 30, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading

rodireich commented Dec 6, 2022 •

edited by github-actions bot

Loading